Skip to content

Conversation

@kasiakoziol
Copy link
Collaborator

@kasiakoziol kasiakoziol commented Jul 28, 2025

Depends on

Description

This PR implements a separation between ingestion and indexing services within the Splunk Operator for Kubernetes. The goal is to enable the operator to independently manage the ingestion service while maintaining seamless integration with the indexing service.

Key Changes

  • Introduction of new Custom Resource Definition for IngestorCluster, Queue and ObjectStorage
  • Support for Remote Queue and Pipeline inputs in IngestorCluster and IndexerCluster via Queue and ObjectStorage reference

Testing and Verification

  • Manual tests
  • Automated tests

Related Issues

Jira Epic: https://splunk.atlassian.net/browse/CSPL-3549

PR Checklist

  • Code changes adhere to the project's coding standards.
  • Relevant unit and integration tests are included.
  • Documentation has been updated accordingly.
  • All tests pass locally.
  • The PR description follows the project's guidelines.

@kasiakoziol kasiakoziol force-pushed the CSPL-3551-ingestion-cr branch 6 times, most recently from ca230a9 to eb21049 Compare July 28, 2025 15:15
@kasiakoziol kasiakoziol force-pushed the CSPL-3551-ingestion-cr branch from eb21049 to 2cca0d7 Compare July 29, 2025 08:24
@coveralls
Copy link
Collaborator

coveralls commented Jul 29, 2025

Pull Request Test Coverage Report for Build 21912466524

Details

  • 549 of 1065 (51.55%) changed or added relevant lines in 15 files are covered.
  • 4 unchanged lines in 1 file lost coverage.
  • Overall coverage decreased (-2.6%) to 83.602%

Changes Missing Coverage Covered Lines Changed/Added Lines %
pkg/splunk/enterprise/types.go 6 10 60.0%
pkg/splunk/enterprise/upgrade.go 0 4 0.0%
pkg/splunk/client/enterprise.go 33 39 84.62%
pkg/splunk/enterprise/objectstorage.go 31 40 77.5%
pkg/splunk/enterprise/queue.go 31 40 77.5%
pkg/splunk/enterprise/util.go 26 54 48.15%
internal/controller/indexercluster_controller.go 2 84 2.38%
internal/controller/ingestorcluster_controller.go 58 146 39.73%
pkg/splunk/enterprise/indexercluster.go 113 242 46.69%
pkg/splunk/enterprise/ingestorcluster.go 234 391 59.85%
Files with Coverage Reduction New Missed Lines %
pkg/splunk/enterprise/indexercluster.go 4 66.85%
Totals Coverage Status
Change from base Build 21866948919: -2.6%
Covered Lines: 11288
Relevant Lines: 13502

💛 - Coveralls

@kasiakoziol kasiakoziol force-pushed the CSPL-3551-ingestion-cr branch from d3779b0 to 44349fa Compare July 29, 2025 14:54
@kasiakoziol kasiakoziol force-pushed the CSPL-3551-ingestion-cr branch 13 times, most recently from 3b2cf1c to 5cad7f9 Compare August 5, 2025 08:49
@github-actions
Copy link
Contributor

github-actions bot commented Feb 10, 2026

CLA Assistant Lite bot CLA Assistant Lite bot All contributors have signed the COC ✍️ ✅

@github-actions
Copy link
Contributor

CLA Assistant Lite bot:
Thank you for your submission, we really appreciate it. Like many open-source projects, we ask that you all sign our Contributor License Agreement before we can accept your contribution. You can sign the CLA by just posting a Pull Request Comment same as the below format.


I have read the CLA Document and I hereby sign the CLA


1 out of 2 committers have signed the CLA.
@kasiakoziol
@Igor-splunk
You can retrigger this bot by commenting recheck in this Pull Request

@Igor-splunk
Copy link
Collaborator

I have read the CLA Document and I hereby sign the CLA

@kasiakoziol
Copy link
Collaborator Author

recheck

@Igor-splunk
Copy link
Collaborator

I have read the Code of Conduct and I hereby accept the Terms

@Igor-splunk
Copy link
Collaborator

recheck

@kasiakoziol kasiakoziol changed the title [DO NOT MERGE BEFORE #1606 and #1610] CSPL-3549 Splunk Operator Enhancement – Ingestion and Indexing Separation CSPL-3549 Splunk Operator Enhancement – Ingestion and Indexing Separation Feb 10, 2026

// If queue is updated
if cr.Spec.QueueRef.Name != "" {
if secretChanged || serviceAccountChanged {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not think we need to worry about changes to the serviceAccount name or role binding itself. IRSA is external to Splunk and operates at the pod level to provide temporary AWS credentials. As long as the pod is correctly annotated and associated with the IAM role, IRSA will continue to function independently of Splunk’s internal logic.

Separately, and unrelated to Ingestion and Indexing separation, we should consider what happens if the AWS secret changes in the SmartStore configuration. That scenario also requires attention. If object storage credentials are updated, we need a well-defined mechanism to ensure the new credentials are picked up safely and consistently.

Given these cases, the restart logic needs to be revisited. Instead of implementing custom coordination logic inside the controller, we should rely on Kubernetes-native mechanisms. For example:

  • Let Kubernetes make eviction decisions based on Pod Disruption Budgets.
  • Use standard StatefulSet rolling restart behavior where appropriate.
  • Ensure that any SmartStore configuration changes are applied via Ansible and/or init containers during pod restart.

The goal should be to keep the operator lightweight and defer restart orchestration to Kubernetes wherever possible, rather than embedding complex restart coordination inside the controller.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SmartStore is not implemented to Index and Ingestion separation, so that's a separate case as mentioned by you. We might want to get back to this topic in the nearest future.

Yes, we need to revisit the restart implementation. I created a task as asked by you and we might want to look into this soon.

@kasiakoziol kasiakoziol force-pushed the CSPL-3551-ingestion-cr branch from 6f08c09 to 7d7a55e Compare February 11, 2026 15:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants